Method For Providing A User Interface Using Three-Dimensional Gestures And An Apparatus Using The Same

ABSTRACT

Provided is a method capable of making various modifications to widgets, graphic objects, or images, which are displayed on a display device, according to motions of a plurality of input units such as finger or stylus pen, with the use of a three-dimensional multi-sensor configured to detect the motions of the input units in a space, without touching the display device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.12/577,978, filed Oct. 13, 2009, now pending, which claims priorityunder 35 U.S.C. 119 and 35 U.S.C. 365 to Korean Patent Application No.10-2008-0099972 (filed on Oct. 13, 2008), both of which are herebyincorporated by reference in its entirety.

BACKGROUND

The present disclosure relates to a user interface controlling method.

More particularly, the present disclosure relates to a method capable ofmaking various modifications to widgets, graphic objects, or images,which are displayed on a display device, according to motions of aplurality of input units such as finger or stylus pen, with the use of athree-dimensional (3D) multi-sensor configured to detect the motions ofthe input units in a space, without touching the display device.

Today, various input/output devices have been mounted on electronicsystems such as TVs, mobile phones, MP3 players, PMPs, notebookcomputers, and PDAs.

Such input/output devices are provided to enable a user to control thosesystems conveniently.

Since portable devices among electronic systems, such as mobile phones,MP3 players, PMPs, notebook computers, or PDAs are small in size, thereis a limitation in mounting input/output devices.

Recently, as one of efforts to improve a user interface, touchscreens,touchpads, or navigation pads have been mounted on portable devices. Theadoption of the touchscreen replaces the input of mouse and keyboard ina general PC, and enables a user to input data and instructions invarious manners even in a small space.

In spite of such various user interfaces, there are still manylimitations in input and output of devices, and users feel inconvenientas much.

In the case of a mobile phone, various additional functions in additionto a call function tend to be added. For example, MP3 player, camerafunction, and Internet access function have been added.

Although it is apparent that those functions are configured to becontrolled by users, there is an inconvenience in controlling thosefunctions appropriately according to use states of devices.

SUMMARY

In one embodiment, a method for controlling a three-dimensional userinterface of a display device includes: displaying one or more widgetsto have predetermined depths on a display of the display device;detecting a user's gesture on a widget selected among the one or morewidgets; and modifying the depths of the widgets.

In another embodiment, a method for controlling a three-dimensional userinterface of a display device includes: displaying one or more widgetsto have predetermined depths on a display of the display device;measuring use frequency of the widgets; and displaying the widgets whilechanging the depths of the widgets according to the use frequency.

In further another embodiment, a method for controlling a user interfaceof a display device using a gesture in a space includes: detecting agesture of selecting a predetermined region of an image displayed on adisplay; detecting a motion of an input unit over the display; andmodifying an image configuration of the selected region according to thedetected motion.

In still further another embodiment, a display device includes: athree-dimensional proximity sensor configured to sense position andmotion of at least one input unit in a space; a memory configured tostore data and program necessary for operations; a display configured todisplay one or more widgets; and a controller configured to control thedisplay to display a widget configuration modified according to themotion of the input unit detected by the three-dimensional proximitysensor, based on the program stored in the memory.

The details of one or more embodiments are set forth in the accompanyingdrawings and the description below. Other features will be apparent fromthe description and drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a mobile terminal according to anembodiment.

FIG. 2 is a configuration diagram of a communication system operablewith the mobile terminal according to an embodiment.

FIGS. 3 to 7 illustrate a method for controlling widgets displayed on amobile terminal 100 using gestures in a space according to anembodiment.

FIGS. 8 to 10 illustrate a method for controlling widgets using aplurality of input units according to an embodiment.

FIGS. 11 to 12 illustrate a method for selecting widgets displayed on atouchscreen using gestures in a space according to an embodiment.

FIGS. 13 to 18 illustrate a method for spatially rotating widgets orgraphic objects displayed on a touchscreen using gestures in a spaceaccording to an embodiment.

FIGS. 19 to 21 illustrate a method for moving widgets displayed on atouchscreen using gestures in a space according to an embodiment.

FIG. 22 illustrates throwing of widgets or graphic objects and collisionfeeling according to an embodiment.

FIGS. 23 to 29 illustrate a method for modifying images using user'sgestures in a space according to an embodiment.

FIGS. 30 to 34 illustrate examples in which the foregoing embodimentsare applied to human being's face images.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Reference will now be made in detail to the embodiments of the presentdisclosure, examples of which are illustrated in the accompanyingdrawings.

FIG. 1 is a block diagram of mobile terminal 100 according to anembodiment.

The mobile terminal 100 may be implemented using a variety of differenttypes of terminals. Examples of such terminals include mobile phones,user equipment, smart phones, computers, digital broadcast terminals,personal digital assistants (PDA), portable multimedia players (PMP) andnavigators.

Although the mobile terminal 100 is exemplarily described below, thefollowing embodiments may be applied to any display devices such asnon-mobile devices monitors and TVs.

By way of non-limiting example only, further description will be withregard to a mobile terminal.

However, such teachings apply equally to other types of terminals. FIG.1 shows the mobile terminal 100 having various components, but it isunderstood that implementing all of the illustrated components is not arequirement. Greater or fewer components may alternatively beimplemented.

FIG. 1 shows a wireless communication unit 110 configured with severalcommonly implemented components. For instance, the wirelesscommunication unit 110 typically includes one or more components whichpermits wireless communication between the mobile terminal 100 and awireless communication system or network within which the mobileterminal is located.

The broadcast receiving module 111 receives a broadcast signal and/orbroadcast associated information from an external broadcast managingentity via a broadcast channel. The broadcast channel may include asatellite channel and a terrestrial channel. The broadcast managingentity refers generally to a system which transmits a broadcast signaland/or broadcast associated information. Examples of broadcastassociated information include information associated with a broadcastchannel, a broadcast program, a broadcast service provider, etc. Forinstance, broadcast associated information may include an electronicprogram guide (EPG) of digital multimedia broadcasting (DMB) andelectronic service guide (ESG) of digital video broadcast-handheld(DVB-H).

The broadcast signal may be implemented as a TV broadcast signal, aradio broadcast signal, and a data broadcast signal, among others. Ifdesired, the broadcast signal may further include a broadcast signalcombined with a TV or radio broadcast signal.

The broadcast receiving module 111 may be configured to receivebroadcast signals transmitted from various types of broadcast systems.By nonlimiting example, such broadcasting systems include digitalmultimedia broadcasting-terrestrial (DMB-T), digital multimediabroadcasting-satellite (DMB-S), digital video broadcast-handheld(DVB-H), the data broadcasting system known as media forward link only(MediaFLO®) and integrated services digital broadcast-terrestrial(ISDB-T). Receiving of multicast signals is also possible. If desired,data received by the broadcast receiving module 111 may be stored in asuitable device, such as memory 160.

The mobile communication module 112 transmits/receives wireless signalsto/from one or more network entities (e.g., base station, Node-B). Suchsignals may represent audio, video, multimedia, control signaling, anddata, among others.

The wireless internet module 113 supports Internet access for the mobileterminal. This module may be internally or externally coupled to theterminal.

The short-range communication module 114 facilitates relativelyshort-range communications. Suitable technologies for implementing thismodule include radio frequency identification (RFID), infrared dataassociation (IrDA), ultra-wideband (UWB), as well at the networkingtechnologies commonly referred to as Bluetooth and ZigBee, to name afew.

Position-location module 115 identifies or otherwise obtains thelocation of the mobile terminal. If desired, this module may beimplemented using global positioning system (GPS) components whichcooperate with associated satellites, network components, andcombinations thereof.

Audio/video (A/V) input unit 120 is configured to provide audio or videosignal input to the mobile terminal. As shown, the A/V input unit 120includes a camera 121 and a microphone 122. The camera receives andprocesses image frames of still pictures or video.

The microphone 122 receives an external audio signal while the portabledevice is in a particular mode, such as phone call mode, recording modeand voice recognition. This audio signal is processed and converted intodigital data. The portable device, and in particular, A/V input unit120, typically includes assorted noise removing algorithms to removenoise generated in the course of receiving the external audio signal.Data generated by the A/V input unit 120 may be stored in memory 160,utilized by output unit 150, or transmitted via one or more modules ofcommunication unit 110. If desired, two or more microphones and/orcameras may be used.

The user input unit 130 generates input data responsive to usermanipulation of an associated input device or devices. Examples of suchdevices include a keypad, a dome switch, a touchpad (e.g., staticpressure/capacitance), a jog wheel and a jog switch. A specific exampleis one in which the user input unit 130 is configured as a touchpad incooperation with a touchscreen display (which will be described in moredetail below).

Also, as described later, the user input unit 130 may include a 3Dproximity sensor configured to detect a user's gesture in a space. The3D proximity sensor is configured to detect two-dimensional (2D) or 3Dposition of an input unit such as a finger or a stylus pen within apredetermined distance, that is, a position (xy coordinates) on display151 and a vertical distance (z coordinate) from the display 151.

Examples of the 3D proximity sensor include an ultrasonic sensor, aninfrared sensor, a capacitive touchscreen, and an image sensor.Technologies to detect position and motion of the input unit in a spaceusing those sensors have been well known. The image sensor may include acamera which takes an image of the input unit.

In this embodiment, an example in which an ultrasonic sensor is used asthe 3D sensor will be described. However, it is apparent to thoseskilled in the art that the invention includes the use of other sensors.

The sensing unit 140 provides status measurements of various aspects ofthe mobile terminal. For instance, the sensing unit may detect anopen/close status of the mobile terminal, relative positioning ofcomponents (e.g., a display and keypad) of the mobile terminal, a changeof position of the mobile terminal or a component of the mobileterminal, a presence or absence of user contact with the mobileterminal, orientation or acceleration/deceleration of the mobileterminal. As an example, consider the mobile terminal 100 beingconfigured as a slide-type mobile terminal. In this configuration, thesensing unit 140 may sense whether a sliding portion of the mobileterminal is open or closed. Other examples include the sensing unit 140sensing the presence or absence of power provided by the power supply190, the presence or absence of a coupling or other connection betweenthe interface unit 170 and an external device.

The interface unit 170 is often implemented to couple the mobileterminal with external devices. Typical external devices includewired/wireless headphones, external chargers, power supplies, storagedevices configured to store data (e.g., audio, video, pictures, etc.),earphones, and microphones, among others. The interface unit 170 may beconfigured using a wired/wireless data port, a card socket (e.g., forcoupling to a memory card, subscriber identity module (SIM) card, useridentity module (UIM) card, removable user identity module (RUIM) card),audio input/output ports and video input/output ports.

The output unit 150 generally includes various components which supportthe output requirements of the mobile terminal. Display 151 is typicallyimplemented to visually display information associated with the mobileterminal 100. For instance, if the mobile terminal is operating in aphone call mode, the display will generally provide a user interface orgraphical user interface which includes information associated withplacing, conducting, and terminating a phone call. As another example,if the mobile terminal 100 is in a video call mode or a photographingmode, the display 151 may additionally or alternatively display imageswhich are associated with these modes.

Also, the display 151 displays a graphic object or an image as describedbelow.

One particular implementation includes the display 151 configured as atouchscreen working in cooperation with an input device, such as atouchpad. This configuration permits the display to function both as anoutput device and an input device.

The display 151 may be implemented using known display technologiesincluding, for example, a liquid crystal display (LCD), a thin filmtransistor-liquid crystal display (TFT-LCD), an organic light-emittingdiode display (OLED), a flexible display and a three-dimensionaldisplay. The mobile terminal may include one or more of such displays.An example of a two-display embodiment is one in which one display isconfigured as an internal display (viewable when the terminal is in anopened position) and a second display configured as an external display(viewable in both the open and closed positions).

FIG. 1 further shows output unit 150 having an audio output module 152which supports the audio output requirements of the mobile terminal 100.The audio output module is often implemented using one or more speakers,buzzers, other audio producing devices, and combinations thereof. Theaudio output module functions in various modes including call-receivingmode, call-placing mode, recording mode, voice recognition mode andbroadcast reception mode. During operation, the audio output module 152outputs audio relating to a particular function (e.g., call received,message received, and errors).

The output unit 150 is further shown having an alarm 153, which iscommonly used to signal or otherwise identify the occurrence of aparticular event associated with the mobile terminal. Typical eventsinclude call received, message received and user input received. Anexample of such output includes the providing of tactile sensations(e.g., vibration) to a user. For instance, the alarm 153 may beconfigured to vibrate responsive to the mobile terminal receiving a callor message. As another example, vibration is provided by alarm 153responsive to receiving user input at the mobile terminal, thusproviding a tactile feedback mechanism. It is understood that thevarious output provided by the components of output unit 150 may beseparately performed, or such output may be performed using anycombination of such components.

A memory 160, as described below, is configured to store data orsoftware necessary for operation of the mobile terminal 100.

The memory 160 is generally used to store various types of data tosupport the processing, control, and storage requirements of the mobileterminal. Examples of such data include program instructions forapplications operating on the mobile terminal, contact data, phonebookdata, messages, pictures, video, etc. The memory 160 shown in FIG. 1 maybe implemented using any type (or combination) of suitable volatile andnon-volatile memory or storage devices including random access memory(RAM), static random access memory (SRAM), electrically erasableprogrammable read-only memory (EEPROM), erasable programmable read-onlymemory (EPROM), programmable read-only memory (PROM), read-only memory(ROM), magnetic memory, flash memory, magnetic or optical disk,card-type memory, or other similar memory or data storage device.

The controller 180 typically controls the overall operations of themobile terminal. For instance, the controller performs the control andprocessing associated with voice calls, data communications, videocalls, camera operations and recording operations. If desired, thecontroller may include a multimedia module 181 which provides multimediaplayback. The multimedia module may be configured as part of thecontroller 180, or this module may be implemented as a separatecomponent.

Also, the controller 180 controls the display 151 to modify and displaya structure of a graphic object or a structure of a selected image area,according to motion of the input unit in a space, which is detected bythe 3D proximity sensor of the user input unit 130.

The power supply 190 provides power required by the various componentsfor the portable device. The provided power may be internal power,external power, or combinations thereof.

Various embodiments described herein may be implemented in acomputer-readable medium using, for example, computer software,hardware, or some combination thereof. For a hardware implementation,the embodiments described herein may be implemented within one or moreapplication specific integrated circuits (ASICs), digital signalprocessors (DSPs), digital signal processing devices (DSPDs),programmable logic devices (PLDs), field programmable gate arrays(FPGAs), processors, controllers, micro-controllers, microprocessors,other electronic units designed to perform the functions describedherein, or a selective combination thereof. In some cases, suchembodiments are implemented by controller 180.

For a software implementation, the embodiments described herein may beimplemented with separate software modules, such as procedures andfunctions, each of which perform one or more of the functions andoperations described herein. The software codes can be implemented witha software application written in any suitable programming language andmay be stored in memory (for example, memory 160), and executed by acontroller or processor (for example, controller 180).

The mobile terminal 100 of FIG. 1 may be configured to operate within acommunication system which transmits data via frames or packets,including both wireless and wireline communication systems, andsatellite-based communication systems. Such communication systemsutilize different air interfaces and/or physical layers.

Examples of such air interfaces utilized by the communication systemsinclude example, frequency division multiple access (FDMA), timedivision multiple access (TDMA), code division multiple access (CDMA),and universal mobile telecommunications system (UMTS), the long termevolution (LTE) of the UMTS, and the global system for mobilecommunications (GSM). By way of non-limiting example only, furtherdescription will relate to a CDMA communication system, but suchteachings apply equally to other system types.

Referring now to FIG. 2, a CDMA wireless communication system is shownhaving a plurality of mobile terminals 100, a plurality of base stations270, base station controllers (BSCs) 275, and a mobile switching center(MSC) 280. The MSC 280 is configured to interface with a conventionalpublic switch telephone network (PSTN) 290. The MSC 280 is alsoconfigured to interface with the BSCs 275. The BSCs 275 are coupled tothe base stations 270 via backhaul lines. The backhaul lines may beconfigured in accordance with any of several known interfaces including,for example, E1/T1, ATM, IP, PPP, Frame Relay, HDSL, ADSL, or xDSL. Itis to be understood that the system may include more than two BSCs 275.

Each base station 270 may include one or more sectors, each sectorhaving an omnidirectional antenna or an antenna pointed in a particulardirection radially away from the base station 270. Alternatively, eachsector may include two antennas for diversity reception. Each basestation 270 may be configured to support a plurality of frequencyassignments, with each frequency assignment having a particular spectrum(e.g., 1.25 MHz, 5 MHz).

The intersection of a sector and frequency assignment may be referred toas a CDMA channel. The base stations 270 may also be referred to as basestation transceiver subsystems (BTSs). In some cases, the term “basestation” may be used to refer collectively to a BSC 275, and one or morebase stations 270. The base stations may also be denoted “cell sites.”Alternatively, individual sectors of a given base station 270 may bereferred to as cell sites.

A terrestrial digital multimedia broadcasting (DMB) transmitter 295 isshown broadcasting to portable terminals 100 operating within thesystem. The broadcast receiving module 111 (FIG. 1) of the portableterminal is typically configured to receive broadcast signalstransmitted by the DMB transmitter 295. Similar arrangements may beimplemented for other types of broadcast and multicast signaling (asdiscussed above).

FIG. 2 further depicts several global positioning system (GPS)satellites 300. Such satellites facilitate locating the position of someor all of the portable terminals 100. Two satellites are depicted, butit is understood that useful positioning information may be obtainedwith greater or fewer satellites. The position-location module 115(FIG. 1) of the portable terminal 100 is typically configured tocooperate with the satellites 300 to obtain desired positioninformation. It is to be appreciated that other types of positiondetection technology, (i.e., location technology that may be used inaddition to or instead of GPS location technology) may alternatively beimplemented. If desired, some or all of the GPS satellites 300 mayalternatively or additionally be configured to provide satellite DMBtransmissions.

During typical operation of the wireless communication system, the basestations 270 receive sets of reverse-link signals from various mobileterminals 100. The mobile terminals 100 are engaging in calls,messaging, and other communications. Each reverse-link signal receivedby a given base station 270 is processed within that base station. Theresulting data is forwarded to an associated BSC 275. The BSC providescall resource allocation and mobility management functionality includingthe orchestration of soft handoffs between base stations 270. The BSCs275 also route the received data to the MSC 280, which providesadditional routing services for interfacing with the PSTN 290.Similarly, the PSTN interfaces with the MSC 280, and the MSC interfaceswith the BSCs 275, which in turn control the base stations 270 totransmit sets of forward-link signals to the mobile terminals 100.

FIGS. 3 to 7 illustrate a method for controlling widgets displayed onthe mobile terminal 100 using gestures in a space according to anembodiment.

FIG. 3 illustrates a mobile phone with a touchscreen as an example ofthe mobile terminal 100.

Touchscreen 31 may be a combination of the display 151 and the userinput unit 130 of FIG. 1 and a touchpad.

A main menu is displayed on the touchscreen 31 of the mobile phone, andbuttons of each menu are displayed with widgets 32. Each of the widgets32 may be implemented with one graphic object. The widgets 32 may bebuttons or shortcut buttons for executing specific applications.

In this embodiment, the widgets 32 are displayed on the touchscreen 31in 3D images so that the widgets 32 have depths. That is, the widgets 32are displayed so that the user can recognize the depths of the widgets32 on the touchscreen.

Three ultrasonic sensors 33 a, 33 b and 33 c are mounted in the vicinityof the touchscreen 31. The ultrasonic sensors 33 a, 33 b and 33 c mayimplement one component of the user input unit 130 in FIG. 1. Anotherultrasonic sensor may be further provided according to embodiments. Theultrasonic sensors 33 a, 33 b and 33 c radiate ultrasonic waves andmeasure time taken for the ultrasonic waves to be reflected by a fingeror a stylus pen and returned back. A controller (not shown) calculates2D coordinates (xy coordinates) and 3D coordinates (z coordinate: thedistance from the touchscreen 31) on the touchscreen 31, based on themeasured time. In case where a plurality of fingers are detected,position of the finger nearest from the touchscreen 31 is detected.

As described above, in addition to the ultrasonic sensors, a capacitivetouchscreen, an infrared sensor, and an image sensor may be used fordetecting the plurality of input units.

As illustrated in FIG. 4, it is assumed that the user selects onewidget, for example, a widget 39 representing a text message menu, withthe input unit, for example, one finger. As described later withreference to FIGS. 11 and 12, the selection of the widget 39 may beperformed by touching one of the widgets 32 or using physical buttons35.

In some embodiments, when the widget 39 is selected, the selected widget39 may be highlighted, or may be displayed to the user visually torepresent the selection of the widget 39 in various manners. Theselection of the widget 39 may be recognized by the user through soundsor vibrations.

If the user moves the finger in a direction perpendicular to thetouchscreen 31, that is, in A or B direction, in such a state that thewidget 39 is selected, the modified depth of the selected widget 39according to the motion of the finger is displayed. For example, whenthe user's finger moves in the A direction, the widget 39 may bedisplayed as if it is recessed from the touchscreen 31 as illustrated inFIG. 5. When the user's finger moves in the B direction, the widget 39may be displayed as it protrudes from the touchscreen 31 as illustratedin FIG. 6. In this case, the user's finger does not touch thetouchscreen 31, but merely moves in an upper space perpendicular to thewidget 39 as illustrated in FIG. 4.

The depth of the widget 39 may be displayed in various manners. Thewidget 39 may be displayed larger than others or shadow may be displayedaround the widget 39, in order to express the protruding widget 39. Thewidget 39 may be displayed smaller than others or shadow may bedisplayed around the widget 39, in order to express the recessed widget39. A variety of known graphic processing technologies may be used toexpress the protruding or recessed widget 39. Any graphic processingmethods may be used only if the graphic object is recognized by the useras if it protrudes or is recessed from the touchscreen 31.

Furthermore, 3D image displaying methods applicable to image displaydevices may be used to express the depth of the widget 39. For example,using a technology disclosed in U.S. Pat. No. 5,084,763, which generatesa stereoscopic vision by combining two signals corresponding to left andright signals, or a technology disclosed in U.S. Pat. No. 5,717,415,which converts a 2D image into a 3D image, the image of the widget 39may be generated in a 2D image and then converted into a 3D image.Moreover, the 3D image may be displayed using a technology disclosed inU.S. Patent Publication No. 2005/0088516, which displays a 2D image anda 3D image simultaneously.

Any methods may be used only if they can express the depth of the widget39.

In some embodiments, a 2D position of the user's finger may berecognized by the user on the touchscreen 31 by displaying the 2Dposition of the user's finger by means of a mark.

Those embodiments may be applied to arbitrary graphic objects displayedon the touchscreen 31, as well as the widgets 32.

In some embodiments, the depths of the widgets 32 may be displayed withseveral discrete levels. For example, the depths of the widgets 32 maybe classified into four levels, that is, four layers. FIG. 7 illustratesthe widgets with different depths. Numbers 1, 2, 3 and 4 on the widgets32 represent the depths of the widgets 32. “1”, “2”, “3”, and “4”correspond to 1 layer, 2 layers, 3 layers, and 4 layers, respectively.In some embodiments, the numbers representing the depths of the widgets32 may not be displayed on the touchscreen 31, or may be displayedblurrily or translucently. Furthermore, the depths of the widgets 32 maybe displayed with colors. That is, if assigning different colors to thedepths of the widgets 32, the user may recognize the depths of thewidgets 32. As such, data quantity to be processed by the controller ofthe mobile terminal 100 may be reduced by distinguishing the depths ofthe widgets 32 with several discrete levels. That is, the ultrasonicsensors 33 a, 33 b and 33 c need not measure exact z coordinates of thefinger, and the data quantity to be processed may be reduced byclassifying the widgets 32 into specific levels only if the detected zcoordinate values are within specific ranges. Moreover, the user maymore easily recognize the depths of the widgets 32 by classifying thedepths of the widgets 32 with several discrete levels.

By displaying a plurality of widgets with different depths asillustrated in FIG. 7, the user may more easily distinguish the widgetsvisually. Widgets that are frequently used may be displayed to be moreprotruding, and widgets that are not frequently used may be displayed tobe more recessed. Therefore, the user may more easily select thefrequently used widgets.

In some embodiments, by recording the use frequencies of the widgets,that is, the user's selection frequencies of the widgets, differentdepths of the widgets may be automatically displayed as illustrated inFIG. 7. Accordingly, the widgets are displayed with the same depths asillustrated in FIG. 3 when the user uses the mobile terminal 100 at thefirst time, and thereafter, different depths as illustrated in FIG. 7may be displayed according to the use of the mobile terminal 100.

FIGS. 8 to 10 illustrate a method for controlling widgets using aplurality of input units according to an embodiment.

Like the embodiments described above with reference to FIGS. 3 to 7,three ultrasonic sensors 33 a, 33 b and 33 c are mounted in the vicinityof the touchscreen 31 of the mobile terminal 100. The ultrasonic sensors33 a, 33 b and 33 c radiate ultrasonic waves and measure time taken forthe ultrasonic waves to be reflected by a finger or a stylus pen andreturned back. A controller (not shown) calculates 2D coordinates (xycoordinates) and 3D coordinates (z coordinate: the distance from thetouchscreen 31) on the touchscreen 31, based on the measured time.

As described above, in addition to the ultrasonic sensor, a capacitivetouchscreen, an infrared sensor, and an image sensor may be used todetect a plurality of input units.

A main menu is displayed on the touchscreen 31 of the mobile terminal100, and each menu is displayed in an icon form. Each icon may beimplemented with a single graphic object.

As illustrated in FIG. 8, the user may make a gesture of pinching thewidget with two input units. For example, the user may make a gesture ofpinching a widget 39 representing a text message menu with two fingers.

The user's fingers do not touch the touchscreen 31, but merely make apinching gesture at a vertically upper position in the vicinity of thewidget 39 in a space as illustrated in FIG. 9. 2D and 3D coordinates ofthe ends of the two fingers may be calculated by the ultrasonic sensors33 a, 33 b and 33 c. When the ends of the fingers are placed on theedges of the widget 39, it may be determined that the widget 39 ispinched.

In some embodiments, when the pinching of the widget 39 is detected, thepinching of the widget 39 may be visually displayed to the user byhighlighting the surrounding of the widget 39 or other methods. The usermay recognize the pinching of the widget 39 through sounds orvibrations. The widget 39 may be selected by pinching the widget 39.

As illustrated in FIG. 9, in such a state that the user pinches thewidget 39, the user may move fingers in a direction perpendicular to thetouchscreen 31 in a space, that is, an A or B direction.

In this case, the ultrasonic sensors 33 a, 33 b, and 33 c may detectmotions of the fingers in a perpendicular direction to the touchscreen31 in such a state that the two fingers are placed at two positions 26 aand 26 b in the vicinity of the widget 39 of FIG. 9.

For example, when the fingers move in the B direction of FIG. 9, thewidget 36 may be displayed to protrude from the touchscreen 31 asillustrated in FIG. 6. When the fingers move in the A direction, thewidget 36 may be displayed to be recessed from the touchscreen 31 asillustrated in FIG. 5.

As illustrated in FIG. 10, marks 41 a and 41 b may be displayed on thetouchscreen 31 in order to display 2D positions of the user's fingers.

Like the embodiments described above with reference to FIGS. 3 to 7,various graphic methods may be applied to display the widgets 36 and 39in a protruding form.

As described above, the depths of the widgets may be expressed byexpanding the sizes of the widgets or adjusting the brightness of thewidgets. Furthermore, 3D technologies may be used to express the widgetsin 3D images.

FIGS. 11 and 12 illustrate a method for selecting widgets displayed onthe touchscreen 31 using gestures in a space according to an embodiment.

As illustrated in FIG. 11, when a plurality of widgets are displayed onthe touchscreen 31 of the mobile terminal 100, the user may touch andselect the desired widgets on the touchscreen 31. Also, when the user'sfinger is placed over a desired widget 43 and stays for a predeterminedtime, the widget 43 placed at the position 45 corresponding to thefinger may be selected. The selected widget 43 may be highlightedrelative to other widgets.

As illustrated in FIG. 12, the widgets may be selected using two or morefingers. The widget 43 may be selected when the user places two fingersat edges of the widget 43, without touching the widget 43, and then apredetermined time passes by. Also, the widget 43 may be selected bytouching the boundary of the widget 43 with two fingers. As describedabove, the widget 43 may be selected by making a gesture of pinching thewidget 43 at positions around the edges of the widget 43. The selectedwidget 43 may be highlighted. By displaying marks 46 a and 46 brepresenting positions of the fingers, the user may recognize thepositions where the fingers are placed on the touchscreen 31.

The above-described widget selecting method may be applied to theselection of arbitrary graphic objects, as well as widgets.

FIGS. 13 to 18 illustrate a method for spatially rotating widgets orgraphic objects displayed on a touchscreen 31 using gestures in a spaceaccording to an embodiment.

As illustrated in FIG. 13, it is assumed that a rectangularparallelepiped widget 50 or graphic object is displayed on thetouchscreen 31 of the mobile terminal 100. the widget 50 may be selectedin the methods described above with reference to FIGS. 11 and 12. Whenthe widget 50 is in a selected state, the finger may rotate in parallelto the touchscreen 31 in a space corresponding to the widget 50, thatis, in a space of a z-axis direction from the widget 50. The rotation ofthe widget 50 around a virtual axis perpendicular to the touchscreen 31,that is, a virtual axis perpendicular to a z-axis may also be displayedaccording to the rotation of the finger.

For example, as illustrated in (a) of FIG. 14, it is assumed that “A”,“B” and “C” represent the front side, the left side, and the bottom sideof the widget 15, respectively. If the finger rotates clockwise in astate of (a) of FIG. 14, the widget 50 rotates clockwise as illustratedin (b) of FIG. 14. Also, if the finger rotates counterclockwise in astate of (a) of FIG. 14, the widget 50 rotates counterclockwise asillustrated in (c) of FIG. 14.

Furthermore, as illustrated in FIG. 15, the widget 50 may rotate whenthe finger rotate in a plane perpendicular to the touchscreen 31, thatis, around a virtual axis parallel to an x-axis, in such a state thatthe widget 50 is selected.

For example, as illustrated in (a) of FIG. 16, it is assumed that “A”,“C” and “D” represent the front side, the bottom side, and the top sideof the widget 50, respectively. If the user rotates the fingercounterclockwise in a plane perpendicular to the touchscreen 31, thatis, around a virtual axis parallel to an x-axis in such a state of (a)of FIG. 16, the widget 50 rotates so that its bottom side C is shown, asillustrated in (b) of FIG. 16. If the user rotates the finger clockwise,the widget 50 rotates so that its top side D is shown.

In addition, the examples illustrated in FIGS. 13 and 15 may becombined. That is, when the user rotates the finger in a tilteddirection, that is, in a direction that is neither perpendicular norparallel to the touchscreen 31, the widget may rotate around an axissloped with respect to the touchscreen 31 according to a rotatingdirection.

In some embodiments, when the user rotates the virtual axisperpendicular to the touchscreen 31 in a horizontal direction in such astate that the widget 50 or graphic object is pinched, as illustrated inFIG. 17, the rotation of the widget 50 around the axis on the display 31corresponding to the vertical axis according to the rotating directionmay be displayed.

Also, in some embodiments, when the user rotates the widget 50 aroundthe virtual axis parallel to the display in such a state that the widget50 is pinched, as illustrated in FIG. 18, the rotation of the widget 50around the axis on the display 31 corresponding to the horizontal axisaccording to the rotating direction may be displayed.

Furthermore, when the user rotates the finger in an arbitrary directionin such a state that the graphic object 30 is pinched, the direction ofthe graphic object 30 may be rotated according to the rotating directionof the finger.

FIGS. 19 to 21 illustrate a method for moving widgets displayed on thetouchscreen 31 using gestures in a space according to an embodiment.

As illustrated in FIG. 19, it is assumed that a widget 50 or graphicobject is displayed on a touchscreen 31.

When the user moves the finger in arrow directions A, B and C in a spacecorresponding to the widget 50, the widget 50 is lifted in the Adirection, moved in the B direction, and put in the C direction. Thatis, as illustrated in FIG. 19, the widget 50 is lifted and then placedat a different position.

Also, when the user selects the widget 50 by a pinching gesture on aspace corresponding to the widget 50 and moves the pinched finger in thearrow directions A, B and C, as illustrated in FIG. 20, the widget 50may be lifted and placed at a different position as illustrated in FIG.21.

Therefore, according to the above-described embodiments, the widget orgraphic displayed on the touchscreen 31 may be moved to a differentposition using gestures in a space, without touching it.

FIG. 22 illustrates throwing of widgets or graphic objects and collisionfeeling according to an embodiment.

As illustrated in FIG. 22, it is assumed that two widgets 60 or 61 orgraphic objects are displayed on the touchscreen 31 of the mobileterminal 100.

The user may select the widget 60 in the above-described manner andflick the selected widget 60 in one direction. When one finger is used,the user may flick one flinger. When two fingers are used, the user maymake a gesture of throwing the pinched widget 60 in one direction.

The ultrasonic sensors 33 a, 33 b and 33 c or 3D proximity sensorsmounted on the mobile terminal 100 may sense positions of the fingerends and detect the flicking gesture or throwing gesture.

For example, when the widget 60 is flicked or thrown in an arrowdirection E, the widget 60 is thrown in an E direction just like awidget 60′. The throwing speed of the widget 60 may be varied accordingto the flicking speed or the throwing gesture speed.

The widget 60 thrown in the E direction may be collided with anotherwidget 61. If the thrown widget 60 is collided with another widget 61,the widget 60 may stop at its position and the widget 61 may again movethe E′ direction. This may be displayed in the same manner as a casewhere an object is thrown and collided with another object in a realspace. That is, the widget 61 may be moved a predetermined distance inthe E′ direction and then stopped.

In some embodiments, if the thrown widget 60 is placed out of the areaof the touchscreen 31, it may be recognized as a command for deletingthe widget 60. In this case, a confirmation message asking the userwhether to delete the widget 60 or not may be outputted and then thewidget may be deleted.

Also, if the widget 61 collided with the thrown widget 60 is placed outof the area of the touchscreen 31, it may be recognized as a command fordeleting the widget 61.

If such an embodiment is applied to a ball-shaped graphic object, a userinterface may be provided which can intuitively manipulate the menu inthe mobile terminal 100. Using such an embodiment, games may beimplemented.

In the above embodiment, the widgets 60 and 61 may be configured so thatthey are collided only when they have the same depth, and they are notcollided when they have different depths.

FIGS. 23 to 29 illustrate a method for modifying images using user'sgestures in a space according to an embodiment.

As illustrated in FIG. 23, it is assumed that a mesh image 61 isdisplayed on the mobile terminal 100. The mesh image 61 is merelyexemplary, and any image may be applied herein.

The user may set an edit region 62 by drawing a closed curve by dragginga finger or a stylus pen on the touchscreen 31 with the touchscreenpanel, as illustrated in FIG. 23. Although the setting of a circularregion 34 is illustrated in FIG. 23, an arbitrary region may be setaccording to the user's dragging.

When the controller of the mobile terminal 100 detects the closed curvedrawn by the user's finger on the touchscreen 31 with the touchscreenpanel, the region inside the closed curve may be recognized as an imageedit region 62.

In another embodiment, in addition to the user's touch and dragging, theimage edit region 62 may be set by gestures in a space. For example, ifthe user draws a closed curve in a space, a region formed by projectingthe closed curve onto the touchscreen 31 may be recognized as the editregion 62.

After setting the edit region 62, if the user pinches positionscorresponding to the vertical upper region of the boundary of the editregion 62 with two fingers, without touching the touchscreen 31, asillustrated in FIG. 24, the 3D proximity sensor of the mobile terminal100 recognizes the gesture, and the controller recognizes that the editregion 62 is pinched. Like the above-described embodiment, the pinchingof the edit region 62 may be recognized by placing the finger over theedge of the edit region 62, or when the pinching gesture over the edgeof the edit region 62 is detected.

In a pinched state, the user may move the finger in a directionperpendicular to the touchscreen 31, that is, the A or B direction.

If the user moves the finger in the B direction, that is, a directionfar away from the touchscreen 31, the center point of the edit region 62may be displayed as if it rises upward from the touchscreen 31 asillustrated in FIG. 25. On the other hand, if the user moves the fingerin the A direction, that is, a direction approaching to the touchscreen31, the center point of the manipulation region 62 may be displayed asif it settles down as illustrated in FIG. 26.

If the gesture illustrated in FIG. 24 is detected, the controller of thedisplay device 30 modifies the configuration of the edit region 62 as ifthe center of the edit region 62 selected by the controller of thedisplay device 30 rises upward or settles down.

In some embodiments, it may be configured to perform the above operationwhen the user moves one finger in the A or B direction in a verticalupper space of the edit region 62, as illustrated in FIG. 24.

The above operation may be implemented in the following manner. An imageto be edited is overlapped on a virtual plane mesh. As the center pointof the edit region selected in the image to be edited moves, specificpositions (vertexes) related to the center point move along the movementof the center point. The vertexes related to the center point may be setas predetermined patterns or may be set adaptively according to images.

In some embodiments, when the rising degree or the settling degree isslight, a bump map may be overlapped on the edit region 62.

Such an image editing method is used in 3D MAXs or Maya, which is aknown 3D image editing tool. A variety of image editing methods areknown, and arbitrary image editing methods may be used herein.

After the edit region 62 is edited in the above manner, the controllerof the mobile terminal 100 generates a new image by overlapping the editregion 62 in the original region.

In another embodiment, the user may edit the image using gestures in aspace. FIGS. 27 to 29 illustrate an image editing method according to anembodiment.

Like the above-described example, it is assumed that a mesh image 61 isdisplayed on the mobile terminal 100.

Like the above-described embodiment, the user may set an edit region 62by drawing a closed curve by dragging a finger or a stylus pen on thetouchscreen 31 with the touchscreen panel.

After setting the edit region 62, if the user pinches positionscorresponding to the vertical upper region of the boundary of the editregion 62 with two fingers, without touching the touchscreen 31, asillustrated in FIG. 27, the 3D proximity sensor of the mobile terminal100 recognizes the gesture, and the controller recognizes that the editregion 62 is pinched.

After that, if the user moves the finger in a horizontal direction tothe touchscreen 31, the edit region 62 may expand in a horizontaldirection to the touchscreen 31.

If the user moves the finger in the D direction of FIG. 27 whilegrasping the edit region 62, the edit region 62 expands in the Ddirection. Herein, the term “expand” means that the image expands onlyin the D direction, as described later.

As illustrated in FIG. 28, when it is assumed that the user moves thefinger from the original position 36 to a position 36′ in a horizontaldirection, a new region 36″ including both the original position 36 andthe moved position 36′ becomes a new edit region 62.

In this case, as illustrated in FIG. 27, images contained in theoriginal edit region 62 expand in a finger moving direction, that is,the D direction of FIG. 25. The expanded region 36″ is overlapped on theoriginal image. The controller of the mobile terminal 100 generates anew image by overlapping the expanded region 36″ on the original image.It may be visually recognized as if the edit region 62 expands in afinger moving direction.

Although the example where the image is modified in a directionperpendicular to the touchscreen 31 has been described with reference toFIGS. 23 to 26 and the example where the image is modified in adirection horizontal to the touchscreen 31 has been described withreference to FIGS. 27 to 29, the invention may also be applied to a casewhere the user moves the image in a direction oblique to the touchscreen31 in such a state that the edit region is pinched, that is, in a casewhere a vertical component and a horizontal component are combined.

That is the case where the user moves the finger in a direction obliqueto the touchscreen 31 in such a state the edit region is pinched, theedit region may be displayed as if it rises upward or settles down in avertical direction and simultaneously expands in a horizontal direction.

FIGS. 30 to 34 illustrate examples in which the above-describedembodiments are applied to human being's face images.

It is assumed that a woman's face 72 is displayed on the mobile terminal100 as illustrated in FIG. 30. Also, it is assumed that the user sets awoman's nose as an edit region 74.

If the user pinches the edit region 74 in a space and moves the fingersin an E direction perpendicular to the touchscreen 31 in the pinchedstate, the center point of the edit region 74 is displayed as if itrises upward from the center or settles down.

For example, when the finger moves far from the touchscreen 31, the nosemay be displayed as if it is sharpened as illustrated in FIG. 32.

Also, if the user sets both cheek regions of the woman's face as theedit regions 76 a and 76 b and moves the fingers in the F and Gdirections in a space, the image expands in the moving direction and isdisplayed as if the cheek regions are contracted as illustrated in FIG.34.

Therefore, the user may edit the images through the intuitivemanipulation, as the user wants, without expert knowledge of imagefiles.

Any reference in this specification to “one embodiment,” “anembodiment,” “example embodiment,” etc., means that a particularfeature, structure, or characteristic described in connection with theembodiment is included in at least one embodiment of the invention. Theappearances of such phrases in various places in the specification arenot necessarily all referring to the same embodiment. Further, when aparticular feature, structure, or characteristic is described inconnection with any embodiment, it is submitted that it is within thepurview of one skilled in the art to effect such feature, structure, orcharacteristic in connection with other ones of the embodiments.

Although embodiments have been described with reference to a number ofillustrative embodiments thereof, it should be understood that numerousother modifications and embodiments can be devised by those skilled inthe art that will fall within the spirit and scope of the principles ofthis disclosure. More particularly, various variations and modificationsare possible in the component parts and/or arrangements of the subjectcombination arrangement within the scope of the disclosure, the drawingsand the appended claims. In addition to variations and modifications inthe component parts and/or arrangements, alternative uses will also beapparent to those skilled in the art.

1. A method for controlling a user interface of a display device using agesture, the method comprising: detecting a gesture that sets a regionof an image displayed on a display, the set region of the image being asubset of the image and a boundary of the set region of the image beingdefined by the gesture; at a time when the region of the image is set,detecting motion of an input unit related to modifying the set region ofthe image; and modifying, by a controller, an image configuration of theset region of the image according to the detected motion.
 2. The methodaccording to claim 1: wherein the display comprises a touchscreen; andwherein detecting a gesture that sets a region of an image displayed ona display comprises detecting, using the touchscreen, a gesture ofdragging a closed curve on the display, the closed curve defining theboundary of the set region of the image.
 3. The method according toclaim 1: wherein detecting motion of an input unit related to modifyingthe set region of the image comprises detecting whether motion of aninput unit is approaching the display or departing from the display; andwherein modifying an image configuration of the set region of the imageaccording to the detected motion comprises: modifying the set region ofthe image to appear to move upward based on detection of the input unitdeparting from the display; and modifying the set region of the image toappear to move downward based on detection of the input unit approachingthe display.
 4. The method according to claim 1: wherein detectingmotion of an input unit related to modifying the set region of the imagecomprises detecting motion of an input unit moving in a directionparallel to the display; and wherein modifying an image configuration ofthe set region of the image according to the detected motion comprisesmodifying the set region of the image to expand in a directioncorresponding to the direction parallel to the display in which theinput unit is detected as moving.
 5. The method according to claim 1,wherein detecting a gesture that sets a region of an image displayed ona display comprises detecting a gesture of drawing a closed curve inspace over the display, the closed curve defining the boundary of theset region of the image.
 6. The method according to claim 5, whereindetecting a gesture of drawing a closed curve in space over the displaycomprises projecting the closed curve onto the display and setting theboundary of the set region of the image based on projecting the closedcurve onto the display.
 7. The method according to claim 1, furthercomprising, after detecting the gesture that sets the region of theimage displayed on the display, detecting a gesture selecting the setregion of the image, wherein modifying an image configuration of the setregion of the image according to the detected motion comprises modifyingan image configuration of the set region of the image according to thedetected motion conditioned on the set region of the image beingselected at a time of detection of the motion of the input unit relatedto modifying the set region of the image.
 8. The method according toclaim 7: wherein detecting a gesture selecting the set region of theimage comprises detecting a gesture of pinching the boundary of the setregion of the image; and wherein modifying an image configuration of theset region of the image according to the detected motion conditioned onthe set region of the image being selected at a time of detection of themotion of the input unit related to modifying the set region of theimage comprises modifying an image configuration of the set region ofthe image according to the detected motion conditioned on the set regionof the image being in a pinched state at a time of detection of themotion of the input unit related to modifying the set region of theimage.
 9. The method according to claim 1, wherein modifying an imageconfiguration of the set region of the image according to the detectedmotion comprises editing the set region of the image according to thedetected motion and generating a new image by overlapping the editedregion onto a corresponding region in the image.
 10. The methodaccording to claim 1, wherein detecting a gesture that sets a region ofan image displayed on a display comprises detecting a first gesture thatsets a first region of an image displayed on a display, the first regionof the image being a subset of the image and a boundary of the firstregion of the image being defined by the first gesture, furthercomprising: detecting a second gesture that sets a second region of theimage displayed on the display, the second region of the image being asubset of the image, a boundary of the second region of the image beingdefined by the second gesture, and the second region of the image beingdifferent than the first region of the image, wherein detecting motionof an input unit related to modifying the set region of the imagecomprises, at a time when the first region of the image and the secondregion of the image are set, detecting motion of an input unit relatedto modifying the first region of the image and the second region of theimage, and wherein modifying an image configuration of the set region ofthe image according to the detected motion comprises modifying an imageconfiguration of both the first region of the image and the secondregion of the image according to the detected motion.
 11. A devicecomprising: a controller; and a memory coupled to the controller havingstored thereon instructions which, when executed by the controller,causes the controller to perform operations comprising: detecting agesture that sets a region of an image displayed on a display, the setregion of the image being a subset of the image and a boundary of theset region of the image being defined by the gesture; at a time when theregion of the image is set, detecting motion of an input unit related tomodifying the set region of the image; and modifying an imageconfiguration of the set region of the image according to the detectedmotion.
 12. The device according to claim 11, further comprising atouchscreen, wherein detecting a gesture that sets a region of an imagedisplayed on a display comprises detecting, using the touchscreen, agesture of dragging a closed curve on the display, the closed curvedefining the boundary of the set region of the image.
 13. The deviceaccording to claim 11: wherein detecting motion of an input unit relatedto modifying the set region of the image comprises detecting whethermotion of an input unit is approaching the display or departing from thedisplay; and wherein modifying an image configuration of the set regionof the image according to the detected motion comprises: modifying theset region of the image to appear to move upward based on detection ofthe input unit departing from the display; and modifying the set regionof the image to appear to move downward based on detection of the inputunit approaching the display.
 14. The device according to claim 11:wherein detecting motion of an input unit related to modifying the setregion of the image comprises detecting motion of an input unit movingin a direction parallel to the display; and wherein modifying an imageconfiguration of the set region of the image according to the detectedmotion comprises modifying the set region of the image to expand in adirection corresponding to the direction parallel to the display inwhich the input unit is detected as moving.
 15. The device according toclaim 11, wherein detecting a gesture that sets a region of an imagedisplayed on a display comprises detecting a gesture of drawing a closedcurve in space over the display, the closed curve defining the boundaryof the set region of the image.
 16. The device according to claim 15,wherein detecting a gesture of drawing a closed curve in space over thedisplay comprises projecting the closed curve onto the display andsetting the boundary of the set region of the image based on projectingthe closed curve onto the display.
 17. The device according to claim 11:wherein the operations further comprise, after detecting the gesturethat sets the region of the image displayed on the display, detecting agesture selecting the set region of the image, wherein modifying animage configuration of the set region of the image according to thedetected motion comprises modifying an image configuration of the setregion of the image according to the detected motion conditioned on theset region of the image being selected at a time of detection of themotion of the input unit related to modifying the set region of theimage.
 18. The device according to claim 17: wherein detecting a gestureselecting the set region of the image comprises detecting a gesture ofpinching the boundary of the set region of the image; and whereinmodifying an image configuration of the set region of the imageaccording to the detected motion conditioned on the set region of theimage being selected at a time of detection of the motion of the inputunit related to modifying the set region of the image comprisesmodifying an image configuration of the set region of the imageaccording to the detected motion conditioned on the set region of theimage being in a pinched state at a time of detection of the motion ofthe input unit related to modifying the set region of the image.
 19. Thedevice according to claim 11, wherein modifying an image configurationof the set region of the image according to the detected motioncomprises editing the set region of the image according to the detectedmotion and generating a new image by overlapping the edited region ontoa corresponding region in the image.
 20. The device according to claim11: wherein detecting a gesture that sets a region of an image displayedon a display comprises detecting a first gesture that sets a firstregion of an image displayed on a display, the first region of the imagebeing a subset of the image and a boundary of the first region of theimage being defined by the first gesture; wherein the operations furthercomprise detecting a second gesture that sets a second region of theimage displayed on the display, the second region of the image being asubset of the image, a boundary of the second region of the image beingdefined by the second gesture, and the second region of the image beingdifferent than the first region of the image; wherein detecting motionof an input unit related to modifying the set region of the imagecomprises, at a time when the first region of the image and the secondregion of the image are set, detecting motion of an input unit relatedto modifying the first region of the image and the second region of theimage, and wherein modifying an image configuration of the set region ofthe image according to the detected motion comprises modifying an imageconfiguration of both the first region of the image and the secondregion of the image according to the detected motion.